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KEN DYKEMA 

Abstract. We show that if A is a Hilbert-space operator, then the set of all 
projections onto hyperinvariant subspaces of A, which is contained in the von Neu- 
mann algebra vN{A) that is generated by A, is independent of the representation 
of vN{A), thought of as an abstract W*-algebra. 

We modify a technique of Foias, Ko, Jung and Pearcy to get a method for finding 
nontrivial hyperinvariant subspaces of certain operators in finite von Neumann 
algebras. 

We introduce the i?-circular operators as a special case of Speicher's B-Gaussian 
operators in free probability theory, and we prove several results about a _B~circular 
operator z, including formulas for the B-valued Cauchy- and R-transforms of z*z. 
We show that a large class of L°°([0, l])-circular operators in finite von Neumann 
algebras have nontrivial hyperinvariant subspaces, and that another large class of 
them can be embedded in the free group factor L(F3). These results generalize 
some of what is known about the quasinilpotent DT-operator. 



1. Introduction 

The invariant subspace problem for operators on Hilbert space and the related 
hyperinvariant subspace problem are both unresolved and are of importance for un- 
derstanding the structure of Hilbert space operators. Let J{ be a Hilbert space and 
let A e B^K) be a bounded operator on "K. A closed subspace "Kq C J{ is said to be 
A-invariant if A^Kq) C JCq. (Throughout this paper, all subspaces will be assumed 
to be closed.) The subspace "Kq is said to be A-hyperinvariant if it is S'-invariant 
whenever 5* G S(IK) commutes with A. Recall that the invariant subspace prob- 
lem asks whether, for J{ infinite dimensional, every A G has an A-invariant 
subspace that is nontrivial (i.e. neither {0} nor J{ itself), and the hyperinvariant 
subspace problem asks whether every A G i3(J{) that is not a scalar multiple of the 
identity has a nontrivial A-hyperinvariant subspace. 

Uffe Haagerup jTTj made a huge advance on the hyperinvariant subspace problem 
for operators in Hi-factors. He proved that if A belongs to a Ili-factor that is 
embeddable in the ultrapower of the hyperfinite Ili-factor and if the Brown 
measure |lj of A is supported on more than one point, then A has a nontrivial 
hj^erinvariant subspace. (He actually proved much more, namely a result on Brown 
measure decomposition by restricting to hyperinvariant subspaces.) It is therefore of 
particular interest to study the hyperinvariant subspace problem for operators whose 
Brown measure has support reduced to a single point. Since the support of the Brown 
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Figure 1. The quasinilpotent DT-operator T 

measure is contained in the spectrum of the operator, quasinilpotent operators in IIi- 
factors are of special interest. The quasinilpotent DT-operator T in the free group 
factor L(F2), from the family of operators defined in 0, was a particularly compelling 
example to study. The operator T can be realized as a limit in *-moments of strictly 
upper triangular random matrices with i.i.d. complex Gaussian entries above the 
diagonal. Alternatively, as was seen in [5, §4], T can be obtained from a semicircular 
element X and a free copy of //°^([0, 1]) by using projections from the latter to cut 
out the upper triangular part of X] for future reference, note that, X may be replaced 
by a circular operator for this procedure. Pictorially, then, we may represent T as 
in Figure n Here the shaded region has weight 1, the unshaded region has weight 0, 
and these weights are used to multiply entries of a Gaussian random matrix, as was 
similarly considered in the self-adjoint case by Shlyakhtenko in and jl6| . 

In |6j, Haagerup and the author proved that T has a one-parameter family of 
nontrivial hyperinvariant subspaces. The proof utilized precise knowledge of certain 
*-moments of T, conjectured in and proved by Sniady [TH], which implies that TT* 
and k{T^{T*ff'^ have the same moments for every G N. It was also shown in jH] 
that these hyperinvariant subspaces can be characterized in terms of the asymptotic 
rate of decay of as n — oo, for vectors ^ in the Hilbert space. 

It is natural to consider more general operators than T, defined also as limits 
of random matrices or, equivalently, in the approach we will take in this paper, by 
cutting a circular operator Z using projections as in P, §4]. Some of these are pictured 
in Figure El where again the shaded regions indicate weight 1 and the unshaded 
regions have weight 0. It is natural to ask whether these operators have nontrivial 
hyperinvariant subspaces. The approach used in P| for T is not presently tenable, 
however; while individual *-moments for these operators can be calculated rather 
easily, a good general formula is lacking; moreover, such special relations between 




Figure 2. Other operators analogous to T 
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moments of TT* and T^{T*Y as mentioned above are unlikely to be found in more 
general settings. 

In this paper, we use another technique to exhibit nontrivial hyperinvariant sub- 
spaces for all operators in a large class generalizing T, (including those pictured in 
Figure El). This technique is an adaptation of one recently found by Foia§, Jung, Ko 
and Pearcy [10 , which they applied to certain quasinilpotent operators Q in B{'K). 
They consider spectral resolutions of Q^{Q*Y acting on vectors Xq G "K. Our modi- 
fication, is, firstly, to take Q in a Ili-factor M. and for xq to take the trace vector in 
the standard representation of M., and, secondly, to consider simultaneously a unital 
subalgebra Af C J\A and the conditional expectations of Q^{Q*Y onto N for positive 
integers k. 

The class of operators we consider are certain 5-circular operators. We intro- 
duce 5-circular operators, which are a special case of Speicher's 5-Gaussian oper- 
ators jni- Examples include the usual circular operator, Shlyakhtenko's generalized 
circular operators J3] , the quasinilpotent DT-operator T and the operators pictured 
in Figure |21 After proving some facts about 5-circular operators, we specialize to 
5-circular operators in tracial von Neumann algebras when B = L°°{[0, 1]). It turns 
out that these are the operators z^, where rj is any finite Borel measure on [0, 1]^ 
whose push-forwards tTj^t] under the coordinate projections 7ri,7r2 : [0,1]^ [0,1] 
are absolutely continuous with respect to Lebesgue measure and have essentially 
bounded Radon-Nikodym derivatives with respect to Lebesgue measure. When rj is 
Lebesgue measure on [0, 1]^, then Zr, is the usual circular operator. When rj is the 
restriction of Lebesgue measure to the upper triangle pictured in Figure HJ then 2;^ 
is the quasinilpotent DT-operator T, while when rj is, for example, the restriction of 
Lebesgue measure to one of the shaded regions depicted in Figure then is the 
corresponding generalization of T described above. We show that z^j has a nontrivial 
hyperinvariant subspace whenever the following three criteria hold: 

(i) T] is supported in the upper triangle {(s,t) | < s < t < 1}; 

(ii) for some < c < d, the restriction of 77 to {{s,t) |c<s<)f:<(i}isr times 
Lebesgue measure, for some r > 0; 

(iii) for some < a < 1, the restriction of r/ to {(s, t) | a < s < t < 1} is less than 
or equal to R times Lebesgue measure, for some R < 00. 

These conditions on rj are illustrated in Figure El (Actually, some weaker conditions 
on rj suffice — see Theorem 15.81 and Figure 01 ) 

We now describe the contents of the rest of the paper. In [J21 we show the well 
known fact that the projection onto an A-hyperinvariant subspace belongs to the von 
Neumann algebra vN[A) generated by A. We then show that, given an element A of 
a W*-algebra Ai, the set of projections in Ai that correspond to A-hyperinvariant 
subspaces is independent of the normal ^-representation of Ai. The proof is tech- 
nically straightforward, but the result is, we believe, conceptually valuable. We also 
give some related examples. In ^ we prove a version of the construction of hyperin- 
variant subspaces from applicable to certain operators in a tracial von Neumann 
algebra. In ^ we introduce B-circular operators and prove several results about 
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Figure 3. Conditions on rj. 



them. In ^ we use the method from ^to construct nontrivial hyperinvariant sub- 
spaces for the operators 2;^ with t] satisfying conditions (i)-(iii) above. In ^ we 
construct 2;^ in L(F3) when r] is absolutely continuous with respect to Lebesgue mea- 
sure on [0, 1]^, using a method analogous to that of [HI §4]. Finally, in ^ we show 
that is quasinilpotent if t] is supported on the upper triangle and is Lebesgue 
absolutely continuous with bounded Radon-Nikodym derivative near the diagonal. 

Acknowledgements. The author is grateful to Carl Pearcy for providing him with 
an early copy of (TU], to Ron Douglas and Carl Pearcy for helpful discussions about 
hyperinvariant subspaces and to Lars Aagaard and Uffe Haagerup for discussions 
pointing to the main idea of the proof of Proposition 14.91 

2. Hyperinvariant subspaces of operators in W*-algebras 

If "Ko is a subspace of and if p : "K ^ !Ko is the projection onto IKq, then "Kq is 
A-invariant if and only if Ap = pAp. (Throughout this paper, all projections will be 
assumed to be self-adjoint.) We will say that a projection p G BCK) is A-invariant 
if p'K is an A-invariant subspace. 

Let Ai C BCK) be a von Neumann algebra. A subspace CKq C J{ is said to be 
affiliated to A4 if the projection p : J-C ^ [Kq onto CKq belongs to 7V1. Of particular 
interest for an operator A G BCK) are A-invariant subspaces that are affiliated to 
the von Neumann algebra vN{A) generated by A. 

The following result is well known and easy to show. 

Proposition 2.1. Given a Hilbert space "K and an operator A G BCK), if O-Cq C "K 
is an A-hyperinvariant subspace, then "Kq is affiliated to the von Neumann algebra 
vN{A) generated by A in BCK). 

Proof. Let p be the projection onto an A-hyperinvariant subspace. Suppose S is in 
the commutant of vN{A). Then S commutes with A, so Sp = pSp. But also S* 
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commutes with A, so S*p = pS*p and pSp = pS. Thus pS = Sp. By von Neumann's 
double commutant theorem, p G vN{A). □ 

However, there may be A-invariant subspaces that are affihated with vN{A) but 
are not A-hyperinvariant, as the following example shows (see also Examples 12.101 
and I2.H1) . Indeed, this is not surprising, because vN{A) incorporates information 
about how A related to its adjoint A*, while the (abstract) lattice of hyperinvariant 
subspaces of A is a similarity invariant. 

Example 2.2. Let A he ^ i j acting on a three-dimensional Hilbert space. Then 

and p = (^oi^j is the projection onto a subspace that is A-invariant and affiliated 

to vN(A), but not i?-invariant, where B = ( o o ). Note that B commutes with 
^ ^ Vo / 

A, and thus the range of p is not A-hyperinvariant. 

It is in any case natural to ask, when p G vN{A) is a projection onto an A- 
hyper invariant subspace and when tt : vN{A) is a normal, faithful *- 

homomorphism, whether the range of Tc{p) in % must be a 7r(A)-hyperinvariant 
subspace. In other words, given an abstract W*-algebra Ad and an element A & A4, 
are the projections onto A-hyperinvariant subspaces the same for all representations 
of A^? 

As is seen below in Theorem 12.51 an affirmative answer to the above question 
follows readily from the classical result that every normal, faithful *-homomorphism 
of a von Neumann algebra is an amplification followed by an induction. 

Lemma 2.3. Let !K and % be nonzero Hilbert spaces and let "Kq be a subspace of 
"K. Take A G B{'K) and consider the operator A J G i3(!K ® %). Then "Kq is 
A-hyperinvariant if and only if O-Cq ® % is {A ® I) -hyperinvariant. 

Proof Suppose 'Kq^X is (A(g)J)-hyperinvariant. If ^ G IKq, S" G i3(IK) and AS = SA, 
then {S /) commutes with (A ® /). Fixing any 77 G 3C, we have (S*^) ® 77 = 
{S (S) /)(^ ® 77) G O-Cq (g) % and, therefore, SC, G IKq. Thus, J-Cq is A-hyperinvariant. 

On the other hand, suppose J-Cq is A-hyper invariant. If G X, let : ^ CK® 3C 
be the map K)(0 = C ® ^- Then for every B G B^K), we have VrjB = (5 ® I)V^. 
Suppose X G B^K ® 3C) commutes with A^ I. Given ?7i, 772 G %, we have 

v*^xv,,A = v;^x{A ® i)v,, = Av;^xv,,, 

and we deduce JCq C n^. Consequently, X(J{o ® X) C [Kq ® X, and JCq ® 

is {A ® /)-hyperinvariant. □ 

Lemma 2.4. Suppose A G B{'K) and IKq ^ is an A-hyperinvariant subspace. Let 
Pq : !K ^ IKo be the projection onto !Ko and suppose E G B^K) is a projection that 
commutes with A and with Pq. Let Ae denote the operator in Bi^E'K) obtained by 
restricting A to E'K. Then E'Kr] JCq AE-hyperinvariant. 
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Proof. Let S G B{E:K) commute with Ae- Let T = SE e B{^). Then AT = TA, 
so T0<o C J{q. But TJ{ C EJ{, so n JCq) = T(J{o) C EJ{ n JCq. Therefore, 

E'K n IKo is A£;-hyperinvariant. □ 

We let Proj(A^) denote the set of all projections, i.e. self-adjoint idempotents, in 
a von Neumann algebra Ai. As promised, the following theorem allows us to speak 
of hyper invar aint projections of an element of a von Neumann algebra, independent 
of representation on Hilbert space. 

Theorem 2.5. Let A4 C i3(!K) be a von Neumann algebra, let A G Ai, let p G 
Proj(A^) and suppose p'K is a A-hyperinvariant. Let ti : M. ^ be any 

normal, faithful * -representation of M.. Then 7r(p)(J{^) is tt{A) -hyperinvariant. 

Proof. By j2l Ch. I, §4, Thm. 3], there is a Hilbert space %, a projection E in the 
commutant oi M ^ Ix in B^K (g) X) and a unitary U : J{,r E^K ® !X) such that 
7c{x) = U*{x®Ix)EU. But then 7r(p)5{^ = U* E{{p'K)®X). Bv LemmalO pJf^X is 
(A®/3(:)^hyperinvariant. From Lemma "we then obtain that E{p'K®%) = £'(J{(S> 
X) n ® 3C) is E{A ® /3c)-hyperinvariant. Therefore, U*{E{p'K ® X)) = 7i{p)^^ 
is f/*(£'(A /g<;))t/-hyperinvariant, i.e. is 7r(y4)-hyperinvariant. □ 

Definition 2.6. Let M he a. W*-algebra, let A G and let p G Proj(A<). We call p 
an A-hyperinvariant projection if h{p)'Kt^ is a 7r( A) -hyper invariant subspace for one 
(and then for all) normal, faithful *-homomorphisms n : A4 ^ BCKt^). 

Remark 2.7. By a result [31 Cor. 1.5] of Douglas and Pearcy, which utilizes work 
of Hoover ^2), if is a von Neumann algebra that can be written as a direct 
sum Ai = Ail © with Ail a (nonzero) finite type I von Neumann algebra, and if 
A G is not a scalar multiple of the identity, then A has a nontrivial hyperinvariant 
projection. 

In light of Theorem 12.51 it stands to reason that there should be representation- 
independent descriptions (whatever that may mean) of the A-hyperinvariant projec- 
tions in vN{A). In that light, it seems natural to ask the following question. 

Question 2.8. Let A be an operator in Hilbert space such that the von Neumann 
algebra vN{A) it generates is a factor not isomorphic to C. If p is a projection in 
vN{A) and if p is S'-invariant for every element S of vN{A) that commutes with A, 
is p necessarily an A-hyperinvariant projection? 

The answer is negative if we do not require vN{A) be be a factor; indeed, the 
projection p from Example 12.21 belongs to the center of vN{A), but fails to be A- 
hyperinvariant. However, as far as the author knows. Question 12.81 is open, (though 
of course if vN(A) is a factor of type I, then the answer is positive). 

In any case. Examples I2.1(JI and 12.111 below show that even when A generates a 
factor of type I or of type Hi, there may be an A-invariant subspace affiliated to the 
factor that is not A-hyperinvariant. 

We need a preparatory, elementary lemma about n x n matrices. Let {cij \ 1 < 
hi < Ti} be a system of matrix units in M„(C). 
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Lemma 2.9. Let n,p E N with p > 2, n > 2p. Let bi, . . . , bn-p be distinct, strictly 
positive numbers. Then there is e > such that whenever 02, . . . , ctp € (0, e) and 



n—p 



k=l 



k=2 



the *-algebra generated by A is all of Mn{C). 
Proof. We may write 



/O •■■ 61 a2 ■■■ ap ■■■ \ 

' 62 * 



A 



\ 



^n — p j 



where the omitted entries are zero. Let 21 denote the *-algebra generated by A. Let 



B — bkCk^k+p + 0'kei,p+k, 



k=l 



k=2 



SO that 



Then 



while 



n—p 



A = B + ^ bkCk^k+p- 

k=p+l 



n—p 



AA* = BB*+ J2 blek,k, 

k=p+l 



BB* = ^ blek,k + ( ^ «fe) ei,i + ^ 

k=l k=2 k=2 

By choosing e sufficiently small, the nonzero eigenvalues of BB* can be forced to be 
arbitrarily close to b\, b\, . . . , b^, respectively. Then we obtain 

e^GSl, {p+l<k<n-p) (1) 

by taking spectral projections of AA*. We have Acp+i^p+i = feiCi^p+i G 21, so 

ei,i,ei,p+i e 21. (2) 

From 

{A - ei^iA)ek+p,k+p = bkek,k+p, {"2 < k <n - p), 
together with ((T)) and Q, we get 

efc,fc+pe2t, {l<k<n-2p). (3) 

Combined with (|H) this yields 

ek,k&% (l<k<n-p). 
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Using 

ek,kA = bkek,k+p, {2 <k <n-p), 
combined with Q, we now get 

efc,fc+pG2t, {l<k<n-p). (4) 

From 

Aek+p,k+p — bkek,k+p = CLk^i^k+p, (2 < < p) 
together with 0, we get 

ei,k+p e 21, {I <k <p) 

and thus also 

ei,k = ei^k+pek+p,k G 21, {2 < k < p) 

and, using (jH), 

ei,A:+2p = Gl,k+P^k+p,k+'ip ^ ^' (1 < ^ < P)- 

Continuing as long as possible, we get 

ei,fe+3p = ei^k+2pek+2p,k+3p G 21, (!</::< min(p, n - 3p)), 
ei,k+4p = ei,fc+3pefc+3p,fc+4p e 21, (1 < A; < iain{p, n - 4p)), 

yielding 

ei,j e 2t, (1 < J < n) 
and2l = M„(C). □ 

Example 2.10. We will find an operator A, generating a finite type 1 factor and having 
an invaraint subspace affiliated to the factor that is not, however, A-hyperinvariant. 
Let a > and consider the upper-triangular 6x6 matrix 

(0 1 a \ 
2 \ 
3 1 

where the omitted entries are all zero. By Lemma ESI for a sufficiently small we have 
vN{A) = Mq{C). But taking the Jordan canonical form, A is similar to 



B 



■0 1 
1 

1 
1 
• 



say A = SBS^^. Thus, the subspace that is the range of the idempotent oper- 
ator S'diag(l, 1, 1, 0, 0, O)^"^ is A-invariant and is affiliated to vN{A), but is not 
A-hyperinvariant, because it is not invariant under SCS'^, where C = (73 og)- 
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Example 2.11. We will find an operator A, generating a type IIi factor and having 
an invaraint subspace affiliated to the factor that is not, however, A-hyperinvariant. 
Let a > and consider the upper-triangular 10 x 10 matrix 



1 a a a 
2 
3 
4 
5 
6 
7 
8 





where again the omitted entries are all zero. Then 



3 4a 5a 6a 
8 
15 
24 
35 
48 


0/ 
/ 



By Lemma [2. 9( for sufficienatly small a we have 

vN{F^) = Mio(C). 



(5) 



Suppose a Ili-factor M. is generated by {u,b}, where m is a unitary satisfying 
V? = 1 and where 6 > and h has spectrum in [1, 1 + e] for some e > to be 
determined later. For example, (see the interpolated free group factors L{Ft) 
for any t G (1, |] have generators with these properties. Let x = ub and consider 
A = F0X e Mio(C) ®M. Let F*F = ^^^^ A^P^, where Pi, . . . , Pn are orthogonal 
projections and Ai, . . . , A„ are the distinct, nonzero eigenvalues of F*F. Then 



A* A = F*F ® b^ 



b\ 



1=1 



If e is small enough, then by taking spectral projections we get Q 
where Q = ^^^i Pi- Therefore, Q b^^ G vN{A) and 

F®u = A{Q ® r^) G vN{A). 



b^ G vN{A), 



But {F ® uf = F^ ®l. From (0), we get Mio(C) O 1 C vN{A). Consequently, 
1 ® -ufo G vN{A), and A generates the Ili-factor Mio(C) ® M. 
However, F is similar in Mio(C) to its Jordan canonical form 



G 



1 
1 



1 



1 

1 
1 



v 



1 
1 




so y4 = F®x is similar in vN{A) = Mio(C)(g'A^ to G^x. Arguing as in Example 12. 101 
we find a subspace that is A-invariant but not A-hyperinvariant and whose projection 
lies in Mio(C) 1 C vN{A). 
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3. A CONSTRUCTION OF HYPERINVARIANT SUBSPACES 

Foia§, Jung, Ko and Pearcy jTHj recently found a technique that constructs nontriv- 
ial hyperinvariant subspaces of some operators on Hilbert space. In this section, we 
adapt their method so that it will apply to certain operators in tracial von Neumann 
algebras. 

Let be a W*-algebra having a normal, faithful, tracial state r. We will consider 
the singular numbers of operators a E Ai with respect to r, which were treated by 
Fack in jH] and by Pack and Kosaki in [^j. Thus, for t G [0,1], the t-th singular 
number of a is 

st{a) = inf{||a(l | p G Proj(A^), r(p) < t}. (6) 

Of course, the singular numbers are highly dependent on the choice of trace r. We 
may write St{a;T) instead of St{a), in order to avoid any confusion. By ^9, 2.2], we 
have 

st{a) = inf{A > | r(l(A,oo)(|a|)) < t}, (7) 

and the infimum is attained. Here, 1(a.oo)(|0'|) denotes the Borel functional calculus, 
so for B C [0, oo) and x G A4, x > 0, 1_b(x) denotes the spectral projection for x 
corresponding to the set B. 

Let A4 be represented on the Hilbert space L^(A^,r) via the Gelfand-Naimark- 
Segal construction. Given a: G A^, we will let x denote the corresponding element 
of L'^{M,t). Suppose C M is a unital W*-subalgebra and let S : M ^ he 
the r-preserving conditional expectation onto A/". As is well known, S is obtained 
by compressing with respect to the projection e : L^(Al,r) L'^{Af,T\j^) onto the 
subspace of L'^{A4,t) that is identified with L^(A/', rf^) by the inclusion M ^ M.. 
In particular 

exe = eS{x) = S{x)e {x G M). (8) 

Theorem 3.1. Let b E A4. Suppose there are integers p >0 and 1 < n{l) < n(2) < 
• ■ ■ and there are 9 G (0, 1) and fik > 0, (k ElSi), so that 

hm — = (9) 

and there are vectors (k ^ L'^i-^ ■•'^\m) such that 

Ck = i[o,^,](^(&"^'^+^(&*)"^'^+^))a (10) 

and converges with respect to the Hilbert-space norm on L'^{M ,t\j^) to a nonzero 
vector ( G L'^{J^, '^\j\f) as k ^ oo. Then b has a nontrivial, hyperinvariant subspace. 

Proof. Por every A; G N, take a sequence {ckj)'fLi in Af so that c^^- converges to Cfc as 
j ^ oo. We may without loss of generality replace Ai by the smallest von Neumann 
algebra such that 6 G Ai, all Ckj G M and S{M) C M. Then M is countably 
generated and L^(Al,r) is separable. Let = se{b'^''^'>Y = sg{b"-^''\b*)"-^^^). Por 
n G N, let En be the projection-valued spectral measure of b^ip*)^] let 



Xk= I \dEn{k){t) G M 

J[Afc,oo) ^ 
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and Vk = {bT^'^^k. Then b'^^'^^yk = K(fc)([Afc, oo)) and 

Wb^^^'^VkWl = (K(fc)([Afe, oo)r, i) = r(E„(fc)([Afc, oo)). 
From ((Tj), we have r(i?„(fc)((A, oo)) > 9 whenever A < A^, so 

9 < inf r(K(fc)((A, oo))) = r(K(fc)([Afc, oo))) < 1. (11) 

Since ||6"''''^^yfc||2 stays bounded as /c ^ oo, by passing to a subsequence, if necessary, 
we may without loss of generahty assume b^^^^y^ converges in the weak topology to 
a vector ^ G L'^{A4,t) as k ^ oo. From (fTTj). we then have (^,1) > 6', so ^ 7^ 0. 
Moreover, we have 



= r('6«W(6*)"W / UEnik){t)) = t{x,) < f . ^^^^ 

Since Cfc £ -^^ (A/", t t_;v') ? using (jH)) and (jTUl) we have 

||(5*)n(fc)+P^^||2 _ (e6"W+P(6*)"W+Pea,a) = (^(&"^^^^''(&*)"^''^^'')Cfc, Cfc) 

= (^(6"('=)+^(6*)"('=)+P)l[o,.,](^:(fe"(')+^(6*)"('=)+^))a,a) (13) 

< /^A:||Ca:||2- 

If G B{L'^{Ai, r)) and if 5* commutes with b, then we have 



{S^, {b*yO = hm {Sb'^^'^yk, {b*ya) = hm {Sy^, (6*)"(^)+^'a)- 

K— >00 fc— ►OO 



But from (fT ^ and (P |l . 

K5ffc,(6*)'^w+^a)i<ii5|iiiCfcii2y|^. 

By hypothesis, this upper bound tends to zero as A; ^ 00. Therefore, we have 

{S^, {brO = 0. (14) 

Clearly, 

V := {5^ I 5 G B{L\M,t)), Sb = bS} 

is a nonzero 6-hyperinvariant subspace. If {b*y( 7^ 0, then by (fT^ . V is nontrivial. 
If, on the other hand, {b*y( = 0, then b has a nonzero cokernel. Since b is not the zero 
operator, it follows that b{L'^{A4,r)) is a nontrivial 6-hj^erinvariant subspace. □ 

We will make use of the following well known result in application of Theorem Kill 

Lemma 3.2. Let M. he a von Neumann with normal, faithful, tracial state t, let 
a e M and q G Proj(7Vl). If < 9 < T{q), then 

se{a] r) > se iqaq- r(g)~Vt^ ). (15) 

T(q) 
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Proof. Suppose p G Proj(A^), r(p) < 9. Then 

r{q A (1 - p)) > r{q) + r(l - p) - 1 > r(g) - 9, 

so 

r(g-g A (1 -p)) ^ _9_ 

and 

||a(l-p)|| > ||a(gA(l-p))|| > \\qaq{q Ml - p))\\. 
This imphes p5|) directly from the definition □ 

4. i?-CIRCULAR ELEMENTS 

Definition 4.1. Let 5 be a unital *-algebra over C. 

(i) A B -valued * -noncommutative probability space is a pair {A, E), where A is a 
unital *-algebra containing S as a unital *-subalgebra (which makes A into 
a bimodule over B) and where E : A B is a 5-bimodule map satisfying 
E{b) = b for all beB. 

(ii) We say {A,E) is a B-valued Banach *-noncommutative probability space if, 
in addition, A is a unital Banach *-algebra, i? is a closed subalgebra of A and 
E is bounded. 

(iii) We say {A, E) is a B -valued C* -noncommutative probability space if, in addi- 
tion, A is a unital C*-algebra, B a C*-subalgebra of A and i^^ is a projection of 
norm 1 onto B. (It follows from ||20j that then E is positive and a 5-bimodule 
map.) 

(iv) We say {A,E) is a B-valued W -noncommutative probability space if, in ad- 
dition, y4 is a unital W*-algebra, B a W*-subalgebra of A and ii^ is a normal 
projection of norm 1 onto B. 

Definition 4.2. Let {A,E) be a i?-valued *-noncommutative probability space. Let 
a : B —>■ B and (3 : B ^ B he C-linear maps. A B -circular element with covariance 
{a, (3) is an element z E A such that the distribution of the pair {z, z*) is 5-Gaussian 
in the sense of jT^l Def. 4.2.3], with covariance determined by 

E{z*bz) = a{b) 
E{zbz*) = (3{b) 
E{zbz) = E{z*bz*) = E{z) = E{z*) = 

for all b E B. In the case that {A, E) is a -B-valued C*-noncommutative probability 
space, we may call z a B -circular operator. 

Examples 4.3. (i) A usual circular operator z with t{z*z) = r is, in the notation 
of Definition 14.21 a C-circular element with covariance (r, r), where here r 
denotes multiplication by r on C. 
(ii) The generalized circular elements £2 + VXil, (0 < A < 1), considered in \TE\ . 
are C-circular with covariance (A, 1), where again the scalars indicate opera- 
tions of multiplication on C. 
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(iii) A DT(5o,c) operator, considered in [S], is L°°{[0, l])-circular, with covariance 
(a, P), where 

aif)ix) = c' r f{t)dt 
Jo 

mix) = c' [ f{t)dt. 

J X 

This was shown in the appendix to [Hj. 

The i?-valued *-moments of a i?-circular operator can be calculated using Spe- 
icher's free cummulant calculus ^H]- This is amounts to the nested evaluation de- 
scribed by Sniady in ^Hl §4.2]. This technique is reviewed below, using the notation 
7r{ - • ■ } for the bracketing of a noncrossing pair partition tt with the multiplicative 
function of free cummulants as in [13j. 

Remark 4.4. With z a 5-circular operator as above, let ri G N, . . . , s{n) G {1, *} 
and 6i, . . . , 6„ G -B. Then 

Eiz'^'^hz^^^h ■ ■ ■ z'^'^K) = ^{^'^'^^1' • • • ' ^'^"^^n} (16) 

7reNC2(n) 

where the sum is over all non-crossing pair partitions vr of {1, . . . , ra} and where the 
quantity 

7rK«6i,...,^^(")6„}, (17) 

which is the bracketing of the cummulants of the pair (2;, z*), is evaluated as described 
below. In particular, the *-moment (fT^ vanishes if n is odd; so let us assume n is 
even. Let 

TT = {{«l, jl}, • • • , {V2, jn/2}}- (18) 

Then the quantity (fTTj) vanishes unless s{ip) 7^ s{jp) for all p G {1, . . . , 71/2}, i.e. 
unless vr pairs only z with z* . Therefore, the >K-moment (fTBj) vanishes if the number 
of j such that s{j) = * differs from the number of j such that s{j) = 1. The 
quantity (fTTjl is evaluated as follows. Without loss of generality take zi = 1 in (fTH)) . 
Then 



7r{z^«6i,...,z^(")64 



( a{hin'{z<'^h,...,z^^^^-'^b,,^^})) 

b,,in"{z-<^^-^%,,+,,...,z-<''\}) if s(l) 

/5(6i(7fV^2^&2,...,^^(^'^-'^&,.-i})) 

6,,(fr"{z^(^i+i)6,,+i,...,z^(")6„}) if s(l) 



(19) 



where vr' is the restriction of vr to {2, ■ ■ ■ ,ji — 1}, renumbered by left translation to 
become an element of NC2(ji — 2), while vf" is the restriction of vr to {ji + 1, . . . , n}, 
renumbered by translation to become an element of NC2(n — ji), and where we set 
_ _ _ ^ z^{h-^)l).^_^j to be 1 if ji = 2 and 7r"{z'^^^+^^bj^+i, . . . , z^^^^fen} to be 1 

if ji = n. 
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For example, if n = 6 and vr = {{1,4}, {2, 3}, {5, 6}}, then 

n{zbi, z*b2, zbs, z*b4, z*b5, zb^} = P{bi{7i'{z*b2, 2;63}))&4a(65)&6 

= Pibia{b2)bs)b4a{b5)b6, 

with n' = {{1,2}}. 

The following basic properties are special instances of Speicher's results [THj . 

Proposition 4.5. Let {A, E) be a B-valued * -noncommutative probability space and 
let z and z' be B-circular elements in {A,E) with covariances (a,/?) and {a', (3'), 
respectively. Suppose z and z' are *-free over B with respect to E. Then: 

(i) z* is B -circular with covariance 

(ii) z + z' is B-circular with covariance (a + a', P + [3'). 

(iii) Let d G B; then d*zd is B-circular with covariance {ad,Pd), where 

adip) = d*a{dbd*)d 
(3d{b) = d*f3{dbd*)d. 

(iv) Suppose p is a self-adjoint idempotent in B; then in the pBp-valued *- 
noncommutative probability space {pAp, E\py^p) , pzp is pBp-circular with co- 
variance {ap,Pp), where ap, Pp : pBp pBp are given by 

ap{b) = pa{b)p 
Pp{b) = pP{b)p. 

Proof. Part (i) is clear from the cummulant calculus. Part (ii) follows from the 
additivity of free cummulants of *-free variables JHl Thm. 4.1.7]. Part (iii) follows 
from 19, Prop. 4.1.10]. Part (iv) follows from part (iii). □ 

Proposition 4.6. Let B be a unital *-algebra, let [A, E) be a B-valued *-probability 
space, let a, P : B ^ B be C-linear maps and let z ^ A. Then z is a B-circular 
operator with covariance {a,P) if and only if 

z = — Xi = Xi, (20) 

where in the notation of [ini Def. 4.2.3], the distribution of the pair Xi,X2 is B 
Gaussian with covariance determined by 

E{xibxi) = {a{b) + P{b))/2 

E{xibx2) =i{p{b) -a{b))/2 

E{x2bxi) = i{a{b) - P{b))/2 

E{x2hx2) = («(6)+/?(6))/2 

E{xi) = E{x2) = 0. 

Proof. From (j^UI) we have xi = ^^|- and X2 = ^^|~- Now the remaining assersions 
follow from multilinearity of -B-valued cummulants. □ 

Compare the following result to Prop. 2.20], from which it follows in light of 
Proposition | 
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Proposition 4.7. Let [A, E) be a B-valued *-noncommutative probability space and 
let z & A be B-circular with covariance {a, (3). Suppose t : B ^ B is a trace on B. 
Then the restriction of t o E to the *-algebra generated by B U {z} is a trace if and 
only if T{a{h)c) = T{[3{c)h) for all b,c E B. 

Notation 4.8. Let {A, E) be a 5-valued Banach *-noncommutative probability space. 
Given x & A, for b E B with sufficiently small, we set 

oo 

G,{b) = J2E{b{xbr). 

n=0 

Note that Gx is related to the Cauchy transform Gx{b) = E{{b — x)~^), as it appears 
for example in pi] or by Gx{b) = Gx{b~^) for b E B invertible with ||&^^|| 
sufficiently small. 

At the heart of the proof of the following result is a scheme for finding the generating 
function of the Catalan numbers, when we recall that the Catalan number C„ = 
(^^) is the number of non-crossing pair partitions of {1, . . . , 2n}. The author is 
indebted to Lars Aagaard and Uffe Haagerup for discussions of this method for the 
quasinilpotent DT-operator. 

Proposition 4.9. Suppose {A,E) is a B-valued Banach *-noncommutative prob- 
ability space and z E A is a B-circular operator with covariance {a, (3). Then for 
b,cE B with ||&||||c|| sufficiently small, we have 

G,,Ub) = b{l-ba{G,,Ac))y' (21) 

= b{l -ba{cil - cPiG,*Ub))r')y'- (22) 

Moreover, the B-valued R-transform of z*cz is given by 

R,*cz{b) = a{c{l-cf3ib))-'). (23) 

Proof. Using cummulants to evaluate ^-moments of z as in Remark |4.4t we have for 
n > 1, 

E{b{z*czb)'') = J2 b{7T{z*c,zb,...,z*c,zb}), 

NC2(2n) 

Any TT G NC2(2n) can be uniquely written as 

TT = {{l,2A;}}U7r'U7r" (24) 

for some k E {1, . . . ,n}, n' E NC2({2, ...,2k-l}) and ir" E NC2({2A; + 1, . . . , 2n}), 
where NC2(5') for a subset 5* C Z is the set of all non-crossing pair partitions of S, 
and where we set NC2(0) = {0}. Moreover, for any k E {1, . . . , n}, the map 

(7r',7r") ^ {{l,2A;}}U7r'U7r" 

is a bijection from NC2({2, ...,2k-l})x NC2({2A; + 1, . . . , 2n}) to {n E NC2(2n) | 
{1, 2k} E rr}. Finally, with vr as in ()24|) . we have 

tt{z*c, zb, . . . , z*c, zb} = a{c{7i'{zb, z*c, . . . ,zb, z* c}))b{n" {z* c, zb, . . . , z*c, zb}), 
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where vr' G NC2(2A; — 2) and tt" G NC2(2?7, — 2k) are obtained from tt' and vr" by left 
shifting by 1 and, respectively, 2k. Therefore, 

n 

E{h{z*czhy) = ha{c{^'{zh, z*c, zb.z*c})) 

fc=i s'eNC2(2fc-2) b(7i"^z*c zb z*c zb}) 



ba{E{c{zbz*cf-^))E{b{z*czby 



\n—k\ 

uayrjycyzuz cj jjrjyuyz czu 

k=l 

So we have 



n=0 n=l fc=l 

= 6 + 6a r ^ E{c{zbz*cy)\ ( Y E{b{z*czby)\ 

= b + ba{G,bz*{c))G,*cz{b). 
Solving yields (PT|). Since is S-circular with covariance a), we have 

G,fc,.(c) = c(l-c/5(G,.,,(6)))^\ 



Plugging this into (j?T|) yields 

By [1% Thm. 4.1.12], which is due to Voiculescu [21] in a slightly different guise, 
the 5-valued R-transform is 

Rz*cz{b)+b~' = K{br\ 

where 

G,,,,{K{b)) = K{G,*Ub)) = b. 

Therefore, 

b = G^^UKib)) = K{b) (1 - K{b)a{c{l - 0/3(6))-^))-^ 

Solving yields 

= + a(c(l - c/5(6))-i), 
and this gives immediately ()23|). □ 
Remark 4.10. The formula (|22j) gives a continued-fraction-type expansion: 

G,,,M = ^ 



1 — ba 



l-cf3 



1-bai ^ 
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When c = 1 and when 6 = G C, we therefore have 



C — Oi 

Proposition 4.11. Let B he a unital C* -algebra and let a, (3 : B ^ B be C 
linear maps. Then a B-circular operator with covariance {a, (3) can be realized in a 
B -valued C*- noncommutative probability space {A,E) if and only if a and (3 are 
completely positive. 

Proof. Necessity follows from the complete positivity of a projection E : A ^ B 
onto a C*-subalgebra, which was proved by Tomiyama [201 • Sufficiency follows from 
results of Speicher [1^] . Indeed, using Proposition 14. 6t complete positivity of a and 
j3 implies that the covariance matrix rj : B ^ M2{B) = B ® M2(C) given by 

where p = | ( J G M2(C), is completely positive; by [121 Thm. 4.3.1], the restric- 
tion of E to the *-algebra 21 generated by {z} U S is positive and by ^3 Rmk. 
4.3.2], the -B-Gaussian random variables with covariance matrix rj can be realized in 
a S-valued C*-noncommutative probability space. □ 

The following exactness result is a direct consequence of Proposition 14.61 ^ Cor. 
2.3] and the fact that exactness passes to C*-subalgebras. 

Proposition 4.12. Let B be an exact C*-algebra, let {A,E) be a B-valued C*- 
noncommutative probability space and let z E A be a B -circular element. Then the 
C* -algebra C*{B U {z}) is exact. 

Lemma 4.13. Let B be a unital C*-algebra and suppose {A.,E) is a B-valued C*- 
noncommutative probability space with E faithful. For x A, we have 

\\x\\ =limsup||E((a;*x)")f/2". 

ra— >oo 

Proof. We may without loss of generality assume A and B are separable. Let be a 
faithful state on B. Then ch o E is a. faithful state on A. Hence 



\x x. 



= lim (0 o E((a;*x)"))i/2" < limsup \\E{{x*xy')\\^/^'' < \\x\ 



□ 



Proposition 4.14. Let B be a unital C* -algebra and suppose {A,E) is a B-valued 
C* -noncommutative probability space with E faithful. Let a and (3 be completely 
positive maps from B to itself. Suppose z E A is a B-circular element with covariance 
{a,P). Then 

max(||«||, my/' < \\z\\ < 2max(||a||, mY^'. (25) 
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Proof. Let K = max(||a||, Using the recursive formula for evaluating the 

bracketing (jl7|) . one sees by induction on n > 1 that 

h{z'^%u ^^('"^62n}|| < ir"max(||6i||, . . . , ||62n||). 
From (fTBj). we therefore get 

< ir"(#NC2(2n)) 

whenever s(l), . . . , s{2n) G {1, *}, where # NC2(ri) is the number of non-crossing pair 
partitions of {l,...,n}. Therefore, \\E{{z* z)"")]] < K''^{^^). Now Lemma Eni and 
the asymptotics of Catalan numbers yield the upper bound in (|^. 
For the lower bound, we have 

\\z*z\\ > 11^(^*^)11 = ||a(l)|| = ||«|| 
\\zz*\\ > \\E(zz*)\\ = 11/3(1)11 = 



□ 

5. Hyperinvariant subspaces for certain L°°([0, 1])-circular operators 

For completeness, we provide a proof of the following well known characterization 
of normal, completely positive maps from L°° {X, fi) to itself, for fi a probability 
measure. This may be compared to ^3 Ex. 2.8], where, however, some conditions 
are different. 

Lemma 5.1. Let ii be a probability measure on a measurable space (X, M). Let 
TT2 '■ X X X X be the coordinate projection 7r2{x,y) = y. Let rj be a finite, positive 
measure on (X x X, M M) and assume that the push-forward measure 712*^ is 
absolutely continuous with respect to jj, and that the Radon-Nikodym derivative '^^^^*^^ 
is bounded. Then there is a (unique) normal, completely positive map 

a, :L°°(X,/i)-.L°°(X,/i) 

such that for all h G L^(X, /i), 



(^vf)iy)Hy)d^i{y) = fix)h{y)dr]{x,y). (26) 

X JxxX 

We may formally write 

{anf){y) = / f{x)v{dx,y). 
Jx 

Conversely, every normal, completely positive map 

a: L'^{X,ij) ^ L'^{X,ij,) (27) 
arises in this way from a measure rj, and 
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Consequently, 



Proof. We have 



\a\ 



\a(l 



X oo 



(29) 



\f{x)h{y)\dr]{x,y) < 



< 



dfi 



\h{y)\dv{x,y) = \\f\Uj IhiyMn^^y) 
d{TT2^r]) 



l%)MMy) = ll/llc 



dfi 



L1(m)- 



So (j26|l uniquely defines an element a,,/ of L°°{X,fi). Cleary the map ar, is positive 
(therefore, completely positive) and normal. 

Conversely, given a normal, completely positive map a as in (j27|) . for Ei,E2 G M 
define 



viE, xE2)= [ (aiiEjMMy)- 

JE2 



(30) 



Using positivity and normality, t] is seen to extend to a finite, positive measure on 
X X X. Finally, TT2*f]{E) = t]{X x E) and from (j30|l we get that 'it2*V is /x-absolutely 
continuous and holds. Now equation follows directly. □ 

If we desire a completely positive map : L°° (X, /i) — > L°° (X, /i) such that 

r(a,(/)^7)=r(//3,((7)) 

for all f,g & L°^(X, /i), where r(-) = J ■ c//i, (cf. Proposition 14. 7|) . then will need 
to satisfy 



{(3^g){x)h{x)dij{x) = / h{x)g{y)dr]{x,y) 

IX JxxX 

for all h G L^(X, yu), and we will need also vti^t] to be absolutely continuous with 
respect to fi and have bounded Radon-Nikodym derivative, where ni : X x X ^ X 
is the other coordinate projection. We may formally write 



iPrj9)ix) = J g{y)r]{x,dy). 

Consider T> = L°°{[0, 1]) with trace r given by integration with respect to Lebesgue 
measure. We will study D-circular operators in a D-valued W*-noncommutative 
probability space {A4,S) such that ro£^ is a normal, faithful, tracial state on A^. In 
light of the above discussion, this class of operators is precisely the class delineated 
below. 

Definition 5.2. If is a finite, Borel measure on [0, 1]^ whose push-forward measures 
7rj^?7 under both coordinate projections tti, 112 : [0, 1]^ — >• [0, 1] are absolutely continu- 
ous with respect to Lebesgue measure and have bounded Radon-Nikodym derivative. 
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let be a "D-circular operator with covariance where, for all h G L^{[0, 1]), 



iavf)iy)Hy)dy = / f{x)h{y)dT]{x,y), 
{Png){x)h{x)dx = / h{x)g{y)dr]{x,y). 

J [0,1] 2 

It follows from Propositions l4.1l] and l4.7l that such a P-circular operator z^^ exists in 
a tracial P-valued C*-noncommutative probability space, and the Gelfand-Naimark- 
Segal construction then yields Zrf in a D-valued W*-noncommutative probability 
space {Ai,S), with t o £ a. normal, faithful, tracial state on M.. We may also use r 
to denote the faithful trace r o £ on Ai, and for a G A4, we let ||a||2 = T(a*a)^/^, as 
usual. 

Lemma 5.3. \\zJ2 = ^/([0, 1]^)^/^ 
Proof. We have 

W^Jl = r{z*Zr,) = T o£{z*Zn) = ^K(l)) = / Ml))iy)dy = t]{[0, 1]^), 

Jo 

where the last equality is from □ 

Notation 5.4. For Borel subsets A of R, we will use 1^ for the characteristic function 
of A, for example as in 1a{S), when applied via the Borel functional calculus to a 
self-adjoint operator S ^ Ai. On the other hand, the notation xa for A C [0,1] 
will be used for the characteristic function of A considered as an element of V = 
L°°([0,1]) C M. 

Lemma 5.5. // A and B are Borel subsets of [0, 1] and if r]{A x B) = 0, then 

XAZr,XB = 0. 

Proof. We have 

T O £{{xAZ^XBTiXAZr^XB)) = t{xb£ iz*XAZr^)) = T{XBar,iXA)) 

XAix)xBiy)dr]{x,y) = r]{A x B) = 0. 

[0,1]2 

□ 

Lemma 5.6. Suppose for some < a < 1, the restriction ofrj to [a, 1] x [0, 1] is ab- 
solutely continuous with respect to Lebesgue measure, has bounded Radon — Nikodym 
derivative and is supported in {{s,t) | a < s < t < 1}. Then there is K > such 
that for a// n e N and all n G [0, ^ ^^7"^ ]> '^^ have 

iio,,mz-{z;r))>XM, 

where 
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Consequently, if we fix -y E [a, 1), then letting ( = (x[7,i])^G -^^("P? Tf©) and letting 

l^n = \ \ 33 
n! 

we have 

for all n E'N. 

Proof. Let H be the Radon-Nikodym derivative of the restriction of t] to [a, 1] x [0, 1] 
with respect to Lebesgue measure and let > be at least as large as the essential 
supremum ||oo of H. Since ri{[a, 1] x [0, a]) = 0, from Lemma 1^31 we have Xla,i]Zri = 
X[a,i]Zr,X[a,i] and consequently, 

S{z;{z;r) > Xla,i]S{z-{z;r) = ^((X[a,iF.)"XKi]«XKi])")- (34) 

Let /„ denote the right-hand-side of (jMj) . with /o = X[a,i]- Then by the nested 
evaluation described in Remark 14.41 we get fn+i = Xla,i]Pri{fn), for all n > 0. We 
therefore have, whenever a < x < 1, 



< fn+i{x) = {(3r,fn){x) = fn{y)H{x,y)dy<K fn{y)dy. 

J a J X 

It follows by induction on n > that 

< fn{x) < K" ^^",""^" , (a < X < 1). 
If < /i < ^"^^7°^" , then 

1[0,.](^««)")) > l[0,.](/n) > 1[0,.]( \, M = X[p,l], 

where ^ = fi, i.e. where (I32j) holds. The remaining assertions follow directly. 

□ 

Lemma 5.7. Let < c < d < 1 and suppose 

r){[d,l] X led]) = = r]{[c,l] X [0,c]). (35) 

Let : [c, d] [0, 1] 6e 0(x) = (x — c)/((i — c) . Let 

V = {d-c)-\(f) X 0)*(r/r[,,rf]2) 

&e the measure on [0, 1]^ that is {d — c)~^ times the push-forward under (f) x cj) of the 
restriction of rj to [c, d] x [c, d] . Then whenever < 9 < d — c and n E N, we have 

se{z-)>s^{z-,). (36) 

Proof. Let pi = X[o,c] and p2 = X[c,d]- From (jHSl) and Lemma |S3 ZnP2 = {pi+P2)zr,P2 
and Zr^pi = piZffPi, so P2Z^P2 = {p2ZnV2)^- Consequently, by Lemma ESI 

se{z-,) > se{{p2Z,p2T- id - cr\T O £) f^^^^J. (37) 
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By Proposition I4.5n v). ^2-2*7^2 is a p2'P^circular element in {p2-Mp2,S\p^j^pJ with 
covariance where for / G P2'D = L°°{[c,d\), a{f) = P2tt,,(/) and /?(/) = 

P2/3,(/). 

Consider the isomorphism (p : L°°([0, 1]) — > L°°([c, d]) given by </>(/) = f o (p. 
Using this identification, we may regard {p2-M.P2,S\p2Mp2) L°°([0, l])-valued 

W*-noncommutative probabihty space and P2ZrjP2 as an L°°{[0, l])-circular operator 
with covariance (0-^ o a o 0, 0-^ o /3 o 0). Let / G L°°([0, 1]) and h G /.^([0, 1]). Then 



((0-^ oao0)/)(t)/i(t)rft = / ((ao0)/)(c+ (rf-c)t)/i(t)rft 







d 



(d-c)-' [ {4>f){x)h{^)d7^{x,y) 
{d-cY^ I if o (P){x){h o (P){y)dr]{x,y) 



J[c,d]^ 

(d-c)-' [ /(s)Mt)rf((0x0),r7)(s,t) 



{a^,f){t)h{t)dt. 

Thus, 0~^oao0 = arf. Similarly, we have 0~^o/?o0 = Hence, p2ZrjP2 is identified 
with Zr^'. Finally, note that the trace {d — c)~^t o S \ ^^j^^^ is equal to r o0~^ °^\p2Mp2' 
and (jnni) follows from (jHTj). □ 

Theorem 5.8. Consider an L'^ {[0,1]) -circular operator z^, as described in Defini- 
tion \5/A Suppose 

(i) for some < a < 1, the restriction oft] to {{s,t) \ a<s<t<l}is less than 
or equal to R times Lebesgue measure, for some R < 00; 

(ii) for some < c < d < 1, the restriction off] to {{s,t) \ c<s<t<d}isr 
times Lebesgue measure for some r > 0; 

(iii) rj vanishes on 

([c,l] X [0,c])U(Kl] X [c,d])U{[a,l] X [0,1]) 

U {(s, t) I c < t < s < c/} U {(s, t) \ a<t<s<a}. 
Then z^i has a nontrivial, hyperinvariant subspace. 

Proof. Note that we may without loss of generality take d < a. The conditions of the 
theorem are illustrated in Figure 01 

By Lemma I^Tl ifO <9<d — a and n G N, then So{z^) > s_e_(2;",), where 

' d—c ' 

rj' is r{d — c) times the Lebesgue measure supported on {{x,y) \ < x < y < 1}. 
Therefore, cf. Examples 14.3( 111). z^/ is a DT((5o, ^r{d — c))-operator, i.e. is \/r{d — c) 
times a DT(5o, l)-operator T. By Sniady's result on ^-moments of T, it follows 
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that (T*)"T" and (^^T*T)"' have the same *-moments. Hence, for any G N and 
< a < 1, 



Hence, 



n 



-n/2 



Sa{{T*T) 



n\l/2 



r((i — c) \ 



n 



By the operator T has trivial kernel (in fact, the distribution of T*T was explicitly 
determined there). Fixing any 9 E (0, (i — c), we get s_e_(T) 7^ 0, and 



for some a > 0. 

We may apply Lemma 15.61 to 2^. Let K be as in that lemma, and choose 7 
sufficiently close to 1 so that K(l — 7) < ^. Then choosing /i„ as in (jH^ and using 
Stirling's formula for n!, we have 



lim sup — ; — -r < lim sup — 



a 



where c„ converges to a strictly postive number. Therefore, Theorem 13.11 applies, 
with p = 0, and yields a nontrivial hyperinvariant subspace for 2;^. □ 

6. L°°([0, 1])-CIRCULAR OPERATORS IN FREE GROUP FACTORS 

In this section, we construct an -L°°([0, l])-circular operator 2^, as in Definition 15. 2( 
inside of a free group factor, when t] is assumed to be absolutely continuous with 
respect to Lebesgue measure. This construction parallels what was done in §4] 
for the quasinilpotent DT-operator. 

c da 1 + 



d 



* 






* 


\ 






* 





\^ R 




^\ 



Figure 4. Conditions on rj from Theorem 15.81 
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As in the previous section, T) will denote L°°([0, 1]) with trace r given by integration 
with respect to Lebesgue measure, {M.,£) will be a "D-valued W*-noncommutative 
probability space such that r o is a normal, faithful, tracial state on M., and 
Zr, will be a P-circular operator in {M.^£) with covariance (a,,,/?^), described by 
equations (jHH). We write r also for the trace t o £. Let H G L^([0, 1]) be the 
Radon-Nikodym derivative of rj with respect to Lebesgue measure. Then 

K/)(t)= [ H{s,t)f{s)ds 

(38) 

iP,fm= I H{t,u)f{u)du. 







Moreover, the push-forward measures {T^.i)^ri of rj under the coordinate projections tti 
and are absolutely continuous with respect to Lebesgue measure and have Radon- 
Nikodym derivatives equal to the coordinate expectations CEi{H) and CE2{H), re- 
spectively, given by 

CE,iH){x)= I H{x,y)dy 

(39) 

CE2{H){y)= [ H{x,y)dx. 



Thus, the assumption on rj from Definition 15.21 is that CEi{H) and CE2{H) are 
essentially bounded, and then from Proposition 14. 141 and equation of Lemma 15. ![ 
we have 

11^,11 < 2max(||CEi(/7)|U, \\CE2{H)\W/^ . (40) 

Definition 6.1. Let w G L°°([0, 1]^). We will say w is in regular block form if w is 
constant on all blocks in the regular nxn lattice superimposed on [0, 1]^, for some n, 
i.e. if there are n G N and Wij G C, (1 < i,j < n), such that w{s,t) = Wij whenever 
^ < s < ^ and < t < ^, for all integers 1 < i, j < n. For specificity, we may 
then say that to is in n x n regular block form. Let a & Ai. Then we set 

n 

M{w, a) = WijPiapj G TVI, 

where Pi = Xrmi ±] ^ T^- 

The following properties are straightforward. 

Lemma 6.2. (a) M(w, a) is independent of the choice of n so long as w is in 
nxn regular block form. 

(b) M{w, (ai + 02) = CM{w, ai) + M{w, 02) for ai, 02 & M and ( & C. 

(c) If w^^\ w^"^^ G L°°([0, 1]^) are both in regular block form, then there n G N 
such that both are in n x n regular block form; for G C, we then have 

M(Cw(^) + w'^^\ a) = CM{w^^\ a) + M{w^^\ a). 
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For the rest of this section, we will suppose that z G is (scalar) circular with 
respect to r and satisfies t{z) = and t{z*z) = 1 and that T> and z are >K-free (over 
C) with respect to r. Therefore, W*{V U {z}) = /.(Fg). 

Lemma 6.3. Let w G L°°([0, 1]^) be in regular block form, let H = and let 

Tj be the Lebesgue-absolutely-continuous measure on [0, 1]^ whose Radon-Nikodym 
derivative is H. Then M{w,z) is V-circular in {M.,S), with covariance {arf.Pr,). 

Proof. For brevity, write a for M{w,z). Suppose w is in n x n regular block form 
with Wij as in Definition 16.11 The equalities 

£{a*fa) = ar,{f) 

S{afa*) = (3,{f) (41) 
S{afa) = S{a*fa*) = S{a) = S{a*) = 
for / G X>, with and as in are easily verified using freeness. For example, 

n n n 

£{a*fa)= w~lWjk£{piZ*pjfpjZpk) = ^Vk^\wjk\'^£{z*Pjfz) 

i,j,k=l k=l j=l 

n n n n rj/ri 

= ^Pk^\wjk?T{j)jf) = ^Pk^lwjkl"^ / f{s)ds = ariif). 
k=i j=i k=i j=i 

Suppose Zi,Z2,--- G M. are (scalar) circular elements such that T{zj) = and 
T{z*Zj) = 1 and the family V, {zi}, {-22}, • • • is *-free with respect to r. Then the 
family (^a\g({zj, z*} U 'D)^'^_^ is free over V with respect to £. Hence, by [T^ Thm. 
4.2.4], letting 

Ofc = {M{w, zi) + --- + M{w, Zk)) 

the pairs (a^, al) converge in X>- valued moments with respect to £ to P-Gaussian 
elements with covariance given by ()4H) as /c — 00. In other words, converges in 
P-valued *-moments to a P-circular element with covariance (a^,/?^). However, by 
Lemma \b.2\ 

Mf Zi + --- + Zk. 

ak = M[w, -j= ). 

But z' := ^'^'^'^^'^ is a (scalar) circular element with t{z') = and t{{z')*z') = 1 and 
with T> and z' *-free. Thus, ak has the same P-valued ^-moments as a itself, so a is 
"D-circular with covariance (a,,,/?^). □ 

Lemma 6.4. Let H G L-'^([0, 1]), H > and assume the coordinate expectations 
CEi{H) and CE2{H) as in (pn|) are essentially bounded. Let rj be the Lebesgue- 
absolutely-continuous measure on [0, 1]^ whose Radon-Nikodym derivative is H. Let 
w = \fB. . Suppose there is a sequence (ly^"'')^;^ in L°°([0, 1]^) such that 

(i) for each n, w^"'^ is in regular block form, 

(ii) lim„_^oo \\w - w('")||i2([o,i]2) = 0, 

(iii) letting H^""^ = both || CEi (//(")) || 00 and H^^s (//(") )|| 00 remam bound- 
ed as n ^ 00. 



26 KEN DYKEMA 

Let a„ = M{w^"'\ z), with z as in Lemma \6. 31 Then an converges in strong -operator 
topology (in the representation of Ai on L^{Ai,T)) to an element of Ai which is a 
V-circular operator with covariance (a^, 



Proof. By Lemma f6. 31 and PU]) . 

||a„|| < 2max(||Ci5;i(if("))|U, ||CE2(i^("^)||oo)'/', 
so ||a„|| remains bounded as n ^ oo. From Lemma f6. 21 we have 

a„-a„ = M(w(")-w('"),2). 

By Lemma 16.31 a„ — is P-circular with covariance corresponding to the measure 
on [0, 1]^ whose Radon-Nikodym derivative is \w^"''' —w^™'^\'^. Thus, from Lemma f5. 31 
we obtain 

Therefore, a„ is Cauchy in L^(A^, r). Since ||a„|| remains bounded, it follows that a„ 
converges in strong-operator topology to an element a of A^. 

It follows, too, that the P-valued ^-moments of a„ converge in strong-operator 
topology to those of a as — > oo, in the sense that 

s.o.t- lim ^(do<(^)di ■ ■ ■ a'^^'^^dk) = S{doa'^^^di ■ ■ ■ a'^'''>dk) 

n— »oo 

for aA\ k & N , do, . . . , dk E V and . . . , s{k) G {1, *}. Therefore, the P-valued free 
cummulants of a„ converge to those of a in strong-operator topology as n — > oo. Let 
rjn be the Lebesgue absolutely continuous measure on [0, 1]^ whose Radon-Nikodym 
derivative is H^"\ By Lemma f6.3[ a„ is D-circular with covariance and it 

follows that a is D-circular with covariance {a,f3), where for / G X', 

«(/) = s.o.t- lim 

n— »oo 

/?(/) = S.o.t- lim /?,„(/). 

n— >oo 

From (j38p . we have 



K„/)(t)= / \w^''\s,t)\'f{s)ds 
Jo 



But 

ll«r,„(/) - a^(/)||Li([o,i]) = / |K„(/) - ar,U'W)\dt 



(|«;(")(s,t)p-Ks,t)r)/(s)rf^ 



dt 



< - k|)||L2([0,l]2)(lk^"^IU2([0,l]2) + ||u7||l2([o,1]2)) 
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and ||j;^2([o,i]2) remains bounded as n ^ oo, while 

_ ?i;||^2([o,i]2) tends to zero. Therefore, a = a^. Similarly, we find jd = jd^. □ 

Theorem 6.5. Let H G -^^^([0, 1]^) have essentially hounded coordinate expectations 
CEi{H) and CE2{H). Let rj be the Lebesgue absolutely continuous measure on [0, 1]^ 
whose Radon-Nikodym derivative is H . Then there is a V-circular operator z^j with 
covariance {arj,Pn) in W*(V U {z}) ^ ^(Fg). 

Proof. Let w = \/H. By Lemma l(i.4| it will suffice to find a sequence {w^"'^)'^^i 
satisfying hypotheses (i)-(iii) listed there. For integers 1 < i, j < n, let 

wlj = / w{x, y)dydx 

J{i~l)/n J{j~l)/n 



and let w'^^HsA) = w'ff^ whenever isA) G [— , -) x \—, Then w^") is in n x n 



Xn) 
ij 

regular block form; i.e. (i) holds. Let us show 

lim «;|U2([o,ip) = 0. (42) 



Let e > 0. There is a continuous function w : [0, 1]^ [0, oo) such that w||i2 < e. 
Let 

ri/n pj/n 



wf^'^ = I I w{x,y)dydx 

'(i-l)/nJ{j-l)/n 



and let w^"-\s,t) = w'lf whenever (s,t) G ^) x [i^, i). Then 

n 

II (n) ~(n)||2 -2 I (") ~(")|2 

But 

pi/n rj/n 

Iw^^"* — wf""*! < / / \w{x.,y) — w{x.,y)\{n?)dydx 

J{i-l)/nJ{j-l)/n 

(j-i/n pj/n \ 1/2 

/ / \w{x,y) - w{x,y)\'^{n'^)dydx] 
J(i-l)/nJ{j-l)/n J 
/ j-i/n pj/n \ 1/2 

= ni / \w{x,y) — w{x,y)\'^dydx 



\J(i-l)/nJ(j~l)/n 

where the second inequality is because {n?)dydx is a probability measure on [— , -) x 
V—,^). Therefore, 

" pi/n rj/n 

ll^yW _ ^{")||22 < ^ / / \w{x,y) - w{x,y)\^dydx = \\w - w\\\2 < . 

i.j=lhi~'^)/nJ{j-l)/n 

By uniform continuity of w, lim„^oo H""^ — '"^'•"^IIl^ = 0, and using the triangle inequal- 
ity, we get II w — w*^") 11^2 < 3e for n sufficiently large. This proves (jl^ . namely that 
hypothesis (ii) holds. 
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Finally, for (iii), letting = |u7(")|2, we wish to show that \\CEi{H^'''>)\\^ and 
\CE2{H^'^^)\\oo remain bounded as n ^ oo. We have, for x G ^), 



But 



so 



/•i/n p/n 

I I w{x,y){n^)dydx 

J(i~-l)/n J[3-^)/n 

(r-i/n pj/n \ 1/2 

/ / \w{x,y)\'^{n^)dydx 

J{i-l)/n J(j-l)/n 

(pi/n pj/n \ 1/2 

/ / \w{x,y)\'^dydx 

J{i-l)/n J{j-l)ln 



i/n p/n 

w{x, y)\'^dydx 



{CE,{H'^-^)){x)<nY^ / / 

,=1 J{i-l)/n J{j^l)/n 



oo ) 



1 = 

pi/n pi 

= n / \w{x,y)\'^dydx 

J{i-l)/n Jo 
pi/n 

= n {CEi{H)){x)dx < \\CE,{H)\\^ 

J(i~l)/n 

and IICEi lU < \\CEi{H)\U. Similarly, we get HC^sl^^")) lU < \\CE2{H)\\ 
and (iii) holds. □ 

7. Some quasinilpotent L°°([0, 1])-circular operators 

As mentioned in the introduction, the existence of nontrivial hyper invariant sub- 
spaces is presently of special interest for quasinilpotent operators in Ili-factors. In 
this section we give sufficient conditions for an L°°([0, l])-circular operator to be 
quasinilpotent. 

Lemma 7.1. Let Zrj be an L°°([0, 1]) -circular operator as in Definition \5.^ and sup- 
pose rj is supported on the set 

{{s,t) I < s < t < 1} 

Let e G (0, 1), let t]^ be the restriction of rj to 

{{s,t) \ 0<s<t<l,t-s<e} 

and let be the corresponding L°°([0, 1]) -circular operator. Then the spectral radius 
r(z^) of is bounded above by ||%||. 



HYPERINVARIANT SUBSPACES FOR SOME B-CIRCULAR OPERATORS 29 

Proof. Let rj'^ = tj — rj^. Then by Proposition I4.5r ii). 2;^ has the same ^-moments as 
w + w', where w and w' are L°°{[0, l])-circular elements with covariances (a^^,/5^J 
and {arf'^, Prj'J, respetively, and where w and w' are *-free over L°°([0, 1]). Given 
r G [0, 1], since 

Ve{[r,l]x[0,r])=0 = r]'^{[r-e,l]x[0,r]), 
from Lemma f5 -St we have 

where of course X[o,r-e] = if r < e. Letting p be the least integer such that pe > 1, 
we therefore have [w'y = and, given integers k{0), k{l), . . . , k{p) > 0, we also have 

^^(o^/^fcd)^/ . . . yjHP-D^'^Hp) = 0. (43) 

Since = ||(w + w')"!!) by distributing {w + w'Y and using (jinj) . for n > p we 
obtain 

n=0 

Therefore, the spectral radius r(z^) = lim„_^oo Ik^lT''") is bounded above by \\w\\ = 

□ 

Proposition 7.2. Let 6e an L'^ {[0,1]) -circular operator as in Definition \5.iA 
Suppose rj is supported on the set 

{{s,t) I < s < t < 1} 

and for some 6 > 0, the restriction off] to 

{{s,t)\0<s<t<l,t~s<6} (44) 

is absolutely continuous with respect to Lebesgue measure and has bounded Radon- 
Nikodym derivative. Then Zr, is quasinilpotent. 

Proof. For < e < 5, let rj^ be as in Lemma (7. II and let be the Radon-Nikodym 
derivative of rj^ with respect to Lebesgue measure on [0, 1]^. In this context, equa- 
tions and pUj) become 

\\z^J < 2max(||CEi(if,)l|oo, \\CE2{H,)\\^y/^ (45) 



where 



/■min(a;+e.l) 

CE,{H,)= / ^ H,ix,y)dy 

J X 

CE2{H,)= f H,{x,y)dx. 



We thus obtain 



\\CEi{H,)\U \\CE2{H,)\\oo < 4He\\oo < eWHsWoo. 

TT^T 1 /2 

Consequently, from (ji^ . ||-2^J| < 2||if<5||oo v^- Letting e — > and applving Lemma l7.ll 
yields r{z^) = 0. □ 
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